1. Field of the Invention
The present invention relates to an image computing and processing apparatus for computing correspondence between images from a plurality of images and for obtaining disparity (depth) corresponding to the depth of an image, and also to an apparatus for synthesizing from the disparity and image data an image as viewed from a designated viewing direction.
2. Related Art of the Invention
For transmission and storage of moving images and multinocular stereoscopic images, it is desired to reduce the enormous amount of information involved. Further, in image capturing and image presentation, if multinocular stereoscopic images can be presented by synthesizing intermediate images from binocular stereoscopic images, the amount of information in image capturing, transmission, and storage can be reduced. To achieve this goal, many attempts have been made to reduce the redundancy of images by obtaining correlations of pixels between images and estimating the motion and depth of images.
Methods of obtaining pixel correlation can be classified broadly into two groups known as gradient methods and block matching methods. These methods have their own advantages and disadvantages. That is, with gradient methods, minute motion and disparity can be estimated with good accuracy, but the accuracy drops if the amount of estimation becomes large. Furthermore, gradient methods are susceptible to noise because of the use of gradients. Moreover, this type of method is disadvantageous in terms of realtime processing since the process involves iterative calculations to correct an estimate obtained from a gradient.
On the other hand, block matching methods can perform estimation with a constant level of accuracy regardless of the magnitude of the amount of estimation, and are resistant to noise. This type of method, however, has the problem that the proper block size differs depending on the magnitude of the luminance gradient and also on the presence or absence of a region containing discontinuities in motion and depth, the proper block size thus being dependent on the distribution of estimation.
Kanade et al. attempted to resolve this problem by performing iterative calculations whereby the block size, position, and disparity are updated using evaluation criteria accounting for luminance gradient, noise, and disparity distributions. ("A Stereo Matching Algorithm with an Adaptive Window: Theory and Experiment," by Takeo Kanade and Masatoshi Okutomi, 1990)
Further, in synthesizing an intermediate image, since image synthesis is performed only using regions where correspondence between images are established, the image is synthesized only for regions where correspondence between images can be obtained.
The above correspondence method, however, has had the problem that the amount of computation required is enormous.
Further, with the prior art method that examines correspondence between images and computes disparity, if objects are displaced from each other in the depthwise direction and a region of the object in the background is hidden from view by the object in the foreground, the correspondence cannot be determined, and it is therefore not possible to compute disparity. In particular, if the object in the foreground is positioned near the imaging device, the hidden area becomes large and the disparity cannot be obtained over a large region. Furthermore, in the prior art, since the resolution at disparity boundaries is determined by the density of correspondence calculations, there is no alternative but to perform calculations with higher density (increased number of times) if the resolution is to be improved.
Furthermore, for regions where correspondence between images is not clear, correspondence cannot be determined and, therefore, images cannot be synthesized.